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and the image data is displayed. When the user specifies an 
area of image to be cut out from the displayed still picture, 
the image data in the specified area is cut out and recorded 
as a cutout image. Each cutout image recorded is displayed 
in the form of an icon. When any of the icons is selected by 
the user, the corresponding cutout image data is read and 
pasted in a part to be changed in the original image data. 
Thus an image can be easily created by user's choice. 

12 Claims, 14 Drawing Sheets 




04/15/2004, EAST Version: 1.4.1 



US 6,677,967 B2 

Page 2 



U.S. PATENT DOCUMENTS 



4,467,349 A 8/1984 Maloomian 348/77 

4,486,774 A 12/1984 Maloomian 348/77 

4,517,658 A 5/1985 Iida 715/520 

4,521,014 A 6/1985 Sitrick 463/31 

4,539,585 A 9/1985 Spackova et al 382/100 

4,541,010 A 9/1985 Alston 348/283 

4.566.263 A 1/1986 Miyazaki et a) 57/276 

4,591,897 A 5/1986 Edelson 348/589 

4,662,635 A 5/1987 Enokian 463/31 

4,684,995 A 8/1987 Baumeister 348/347 

4,688,105 A * 8/1987 Bloch et al 386/38 

4,710,873 A 12/1987 Breslow et al 463/31 

4,731,743 A 3/1988 Blancato 345/632 

4,803,554 A 2/1989 Pape 348/231.6 

4,823,285 A 4/1989 Blancato 345/630 

4,827,347 A 5/1989 Bell 348/333.05 

4,885,702 A • 12/1989 Ohba 345/419 

5,016,107 A 5/1991 Sasson et al 348/231.1 

5,093,731 A 3/1992 Watanabe et al 386/118 

5,095,798 A 3/1992 Okada et al 463/35 

5,111,283 A 5/1992 Nagasawa et al 348/231.3 

5,111,409 A 5/1992 Gasper et a 1 715/500.1 

5,153,729 A 10/1992 Saito 348/231.6 

5.182.647 A 1/1993 Chang 348/298 

5,184,830 A 2/1993 Okada et al 463/29 

5,185,665 A 2/1993 Okura et al 348/584 

5.185.818 A 2/1993 Warnock 382/U2 

5,191,645 A 3/1993 Carlucci et al 345/723 

5.200.863 A 4/1993 Orii 386/118 

5.237.648 A 8/1993 Mills et al 345^723 

5,239,419 A 8/1993 Kim 386/52 

5.249.242 A 9/1993 Hanson et al 382/269 

5,255,357 A 10/1993 Byron et al 345/170 

5,262,867 A 11/1993 Kojima 348/39 

5,293,236 A 3/1994 Adachi et al 348/231.2 

5,301,026 A 4/1994 Lee 348/584 

5,301,267 A 4/1994 Hassett et al 345/469 

5,303,334 A 4/1994 Snyder et al 358/1.9 

5.305.118 A 4/1994 Schiller et al 358/3.07 

5.343.243 A 8/1994 Maeda 348/22Z1 

5.363.119 A 11/1994 Snyder et al 345/668 

5,369,736 A * 11/1994 Kato et al 345/425 

5,371,540 A 12/1994 Tamura et al 348/2221 

5,396,225 A 3/1995 Okada et al 463/40 

5,414,444 A 5/1995 Britz 345/156 

5,423,554 A * 6/1995 Davis 463/31 X 

5,430,496 A 7/1995 Silverbrook 348/589 

5,437,008 A 7/1995 Gay et al 715/502 

5,438,359 A 8/1995 Aoki 348/231.9 

5,440,401 A 8/1995 Parulski et al 386/124 

5,442,543 A 8/1995 Tresp 700/29 

5.459.819 A 10/1995 Watkins et al 358/1.18 

5,469,536 A * 11/1995 Blank 345/594 

5,475,441 A 12/1995 Parulski et al 348/552 

5,475,539 A 12/1995 Orii 386/118 

5.477.264 A 12/1995 Sarbadhikari et al. ... 348/231.6 

5,493,409 A 2/1996 Maeda et al 358/296 

5,504,842 A 4/1996 Gentile 358/1.15 

5,506,944 A 4/1996 Gentile 358/1.15 

5,519,826 A 5/1996 Harper et al 345/473 

5,524,194 A 6/1996 Chida et al 345/753 

5,533,181 A 7/1996 Bergsneider 345/473 

5,539,459 A 7/1996 Bullitt et al 348/254 

5,539,865 A 7/1996 Gentile 358/1.16 

5,543,835 A 8/1996 Mumura 348/207.99 

5,544,290 A $/1996 Gentile 358/1.16 

5,546,528 A 8/1996 Johnston 345/807 

5.553.864 A 9/1996 Sitrick 463/31 

5,577,189 A 11/1996 Gay et al 345/800 

5,581,299 A 12/1996 Raney 348/64 



5,581,311 A 12/1996 Kuroiwa 348/231.2 

5,586,238 A 12/1996 Murata 345/629 

5,592,609 A 1/1997 Suzuki et al 345/473 

5,595,389 A 1/1997 Parulski et al 463/31 

5,600,767 A 2/1997 Kakiyama et al 345/629 

5,601,487 A 2/1997 Oshima et al 463/4 

5,602,976 A 2/1997 Cooper et al 358/1.17 

5,612,716 A ♦ 3/1997 Chida et al 345/636 X 

5.612.732 A 3/1997 Yuyama et al 348/14.01 

5.621.431 A 4/1997 Harper et al 345/473 

5,625,711 A 4/1997 Nicholson et al 382/224 

5,625,716 A 4/1997 Borg 382/254 

5,631,701 A 5/1997 Miyake 348/222.1 

5,633,678 A 5/1997 Parulski et al 348/231.5 

5.633.733 A 5/1997 Miyazawa 358/527 

5,633,985 A 5/1997 Severson et al 704/267 

5.634.064 A 5/1997 Warnock et a 1 715/513 

5,638,498 A 6/1997 Tyler et al 358/1.18 

5.638.502 A 6/1997 Murata 345/634 

5.638.503 A 6/1997 Hoel 345/442 

5.659.490 A * 8/1997 Imamura 345/626 X 

5,666,159 A 9/1997 Parulski et al 348/211.2 

5,680,533 A 10/1997 Yamato et al 345/473 

5,681,223 A 10/1997 Weinreich 472/61 

5.682.197 A 10/1997 Moghadam et al 348/36 

5,689,611 A 11/1997 Ohta et al 386/46 

5,696,850 A 12/1997 Parulski et al 382/261 

5,706,097 A 1/1998 Schelling et al 358/296 

5,706,457 A 1/1998 Dwyer et al 345/835 

5,708,883 A 1/1998 Segan et al 396/339 

5,715,486 A 2/1998 Kim et al 396/299 

5,719,799 A 2/1998 Isashi 708/105 

5,729,637 A 3/1998 Nicholson et al 382/282 

5,737,452 A 4/1998 Schiller 382/270 

5.737.491 A 4/1998 Allen et al 704/270 

5,742,291 A * 4/1998 Palm 345/646 X 

5,745,122 A 4/1998 Gay et al 345/619 

5,748,196 A ♦ 5/1998 Coelho et al 345/716 

5,748,326 A 5/1998 Thompson-Bell et a). .. 358/296 

5,754,227 A 5/1998 Fukuoka 348/231.6 

5.781.198 A 7/1998 Korn 345/634 

5,784,525 A 7/1998 Bell 386/107 

5,796,429 A 8/1998 Suzuki et al 348/239 

5.830.065 A * 11/1998 Sitrick 463/31 

5,835,101 A * 11/1998 Otsuka 345/539 

5,880,740 A * 3/1999 Halliday et al 345/435 

5,937,081 A * 8/1999 O'Brill et al 382/11 

5,984,780 A * 11/1999 Takemoto et al 463/31 X 

6,026,215 A * 2/2000 Fantone et al 345/435 X 

6,061,532 A 5/2000 Bell 396/661 

6,064,393 A ♦ 5/2000 Lengyel et al 345/426 X 

6,115,036 A 9/2000 Yamato et al 345/723 

6,120,379 A 9/2000 Tanaka et al 463/44 

6.139.432 A 10/2000 Watanabe et al 463/31 

6,208,347 Bl * 3/2001 Migdal et al 345/419 

6,227,974 Bl * 5/2001 Eilat et al 345/419 X 

6,229,904 Bl * 5/2001 Huang et al 345/646 X 

6,256,047 Bl ♦ 7/2001 Isobe et al 345/473 

6,280,323 Bl • 8/2001 Yamazaki et al 463/31 X 

6,282,362 Bl * 8/2001 Murphy et al 358/909.1 X 

6,283,858 Bl * 9/2001 Hayes, Jr. et al 463/31 

6,343,987 B2 * 2/2002 Hayama et al 643/31 X 



OTHER PUBLICATIONS 

Public Patent Bulletin SHO Publication No. 62-00167 
(Application No. 60-139352), Hitoshi, "Half Tone Record- 
ing Device", Jan. 6, 1987, pp. 1-6 plus English Abstract JP 
62000167. 



04/15/2004, EAST version: 1.4.1 



US 6,677,967 B2 

Page 3 



Public Patent Bulletin SHO Publication No. 63-272261 
(Application No. 62-107832), Tomokage, "Picture Informa- 
tion Reader", Nov. 9, 1988, pp. 1-6 plus English Abstract JP 
63272261. 

Public Patent Bulletin HEI Publication No. 02-039195 
(Application No. 63-188421), Koji et al., "Display System 
for Multi- Valued Image", Feb. 8, 1990, pp. 1-7 plus English 
Abstract JP 2039195. 

Public Patent Bulletin SHO Publication No. 58-036076 
(Application No. 56-133863), Masao, "Video Synthesizing 
Device", Mar. 2, 1983, pp. 1-7 plus English Abstract JP 
58036076. 

Public Patent Bulletin SHO Publication No. 63-213078 
(Application No. 62-045991), Kenji et ah, "Picture Synthe- 
sizing Method", Sep. 5, 1988, pp. 1-5 plus English Abstract 
JP 63213078. 

Public Patent Bulletin HEI Publication No. 03-139064 
(Application No. 01-275816), Kauhiro et al., "Image Syn- 
thesizer", Jun. 13, 1991, pp. 1-5 plus English Abstract JP 
3139064. 

Public Patent Bulletin HEI Publication No. 03-275092 
(Application No. 02-074888), Hirobumi et al., "TV Game 
Device", Dec. 5, 1991, pp. 1-5 plus English Abstract JP 
3275092, 

Public Patent Bulletin HEI Publication No. 04-329758 
(Application No. 03-128432), Tomoaki et al, "Picture 
Recorder", Nov. 18, 1992, pp. 1-4 plus English Abstract JP 
4329758. 

Public Patent Bulletin HEI Publication No. 09-140936 
(Application No. 07-303965), Isamu et al., "Game 
Machine", Jun, 3, 1997, pp. 1-8 plus English Abstract JP 
9140936. 

Public Patent Bulletin Publication No. 6-96180 (Applica- 
tion No. EP 19930304530 19930610), Gay et al., "Method 
of Establishing Constraints Between Graphical Elements", 
Dec. 29, 1993, pp. 1-31 plus English Abstract (EP0576178 
A3) JP 6096180 A. 

Public Patent Bulletin Publication No. 5-110835 (Applica- 
tion No. EP19910309007 19911002), Schiller, "Method of 
Assigning Pixels to Cells of a Halftone Grid", Aug. 12, 
1992, pp. 1-23 plus English Abstract JP 5110835. 

Public Patent Bulletin Publication No. 7-46462 (Applica- 
tion No. EP 19930302488 19930330), Ubillos, "Method and 
Apparatus for Video Editing", Oct. 6, 1993, pp. 1-13 plus 
English Abstract JP 7046462A. 

Public Patent Bulletin Publication No. 8-190504 (Applica- 
tion No. EP19950304792 19950710), Paknad et al., 
"Method and Apparatus for Finding File Paths on a Com- 
puter System", Apr. 10, 1996, pp. 1-28 plus English Abstract 
JP 8190504. 

Public Patent Bulletin Publication No. 8-194697 (Applica- 
tion No. EP19950305330 19950731), Ayers, "Method and 
Apparatus for Identifying Words Described in a Page 
Description Language File", Mar. 13, 1996, pp. 1-26 plus 
English Abstract JP8194697A. 

Public Patent Bulletin Publication No. 8-202337 (Applica- 
tion No. EP19950304971 19950717), Dowling et al., 
"Method of Generating Multiple-Master Typefaces", Mar. 
13, 1996, pp. 1-6 plus English Abstract JP8202337. 



Public Patent Bulletin Publication No. 8-249329 (Applica- 
tion No. EP 19950303939 19950608), Ayers et al., "Method 
and Apparatus for Identifying Words Described in a Portable 
Electronic Document", Mar. 20, 1996, pp. 1-26 plus English 
Abstract JP 8249329A. 

Public Patent Bulletin Publication No. 9-282474 (Applica- 
tion No. JP19960115615 19960412), Fuminori, "Grada- 
tional Image Generating Method", Oct. 31, 1997, pp. 1-8 
plus English Abstract JP9282474A. 

Public Util. Mod. Bull. HEI Publication No. 6-14215 
(Application No. JP19910305002 19911120), Koike, "Digi- 
tal Signal Processor", Jan. 21, 1994, pp. 1-5 plus English 
Abstract JP6014215. 

Campbell, Colin, "Digimask Puts Face on Gaming's 

Future," Games Business (May 15, 2000). 

"ECTS: Digimask Faces Gaming Partners," The Fastest 

Game News Online (Sep. 5, 2000). 

"ARIS Builds Digjmast Future," www.uk.aris.com (Aug. 

10, 2000). 

Sturgeon, Hannah, 3-d,netimperative.com (Jul. 30, 2000). 

Wapshott, Tim, "Face up to reality, " The Times Interface 
2000, www.Times-archive.co.uk (Oct. 24, 2000). 

Fudge, James, "Digimask Demonstration Released," (Jun. 
30, 2000). 

Owen, Steve, "Old Telstar Crew Shows Its Face," www- 
.gamcspot.co.uk (May 26, 2000). 

"Facing the Future, Industry: A new technology called 
Digimask allows web users to create 3D animations of their 
own faces," www.fgonline.com, (Mar. 29, 2000). 
"Didgmask puts you in picture," www.Sunday-times.co.uk 
(Mar. 26, 2000). 

Fudge, James, "Digimask at E3Expo 2000," www.cdmag- 
.com (May 3, 2000). 

Press release, "Tomorrow's Not Just Another Day For 
Digimask," www.digimask.com (Oct. 6, 2000). 

Press release, "Digimask Puts Face to Big Game Names," 
www.digimask.com (Sep. 4, 2000). 
Press release, "Digimask Prepares to Face its Public," www- 
.digimask.com (Jul. 21, 2000). 

Press release, "Digimask Appoint Foresight for Web 
Relaunch," www.digimask.com (Jun. 21, 2000). 

Press release, "Pearce Joins Digimask for New Global 
Role," www.digimask.com (Sep. 4, 2000). 

Press release, "Hetherington Secures £5 Million Funding for 
Digimask," www.digimask.com (Mar. 31, 2000). 

Web site information, ign64-ign.com, "Rare Puts Gamers in 
the Game" (Mar. 12, 1999). 

Web site information, ign64-ign.com, "Perfect Dark Ditches 
Face Mapping" (Feb. 2, 2000). 

Web site information, ign64-ign.com, "IGN's Best of E 3 
Awards" Mar. 17, 1999). 

Gelmis, "Nintendo Users Putting on a Game Face," News- 
day (Oct. 13, 1999). 

Web site information, digimask.com. 

GAME BOY® Camera Instruction Booklet. 

* cited by examiner 



04/15/2004, EAST Version: 1.4.1 



U.S. Patent Jan. 13, 2004 Sheet 1 of 14 



US 6,677,967 B2 



FIG. 1 

50 




04/15/2004, EAST Version: 1.4.1 



U.S. Patent Jan. 13, 2004 Sheet 2 of 14 US 6,677,967 B2 



F I G. 2 



10 



GAME MACHINE 
RCP 




..14 





r 






RSP 






RAM 









143 



RDP 



15 



BUS CONTROL 
CIRCUIT 

—r 

141 



CPU 



13 



16 
/ 



CONTROLLER 

CONTROL 

CIRCUIT 



121 



CONNECTOR 



H 



20 



CONTROLLER 



124 

JL 



CONNECTOR 



H 



20 

2_ 



CONTROLLER 



16a 



CONNECTOR 



16v 



CONNECTOR 



DISPLAY 



52 



04/15/2004, EAST Version: 1.4.1 



U.S. Patent Jan. 13, 2004 Sheet 3 of 14 US 6,677,967 B2 



F I G. 3 



311. 



312. 



313, 



R0M31<^ 



314, 



315, 



316, 



RAM32- 



STORAGE REGION MAP IN EXTERNAL STORAGE 
MEDIUM(30,45) 



ID INFORMATION STORAGE REGION 

• SERIAL No. 

• GAME VERSION 



OS INFORMATION STORAGE REGION 

• GRAPHIC MICROCODE 

• SOUND MICROCODE 

• CPU LIBRARY etc. 



IMAGE CREATING PROGRAM STORAGE REGION 
WRITE/READ/ DISPLAY CONTROL PROGRAM 
POLYGON COORDINATE CALCULATING PROGRAM 
ICON DISPLAY PROGRAM 
PARTS SELECT/DISPLAY PROGRAM 
COLOR/BRIGHTNESS CONTROL PROGRAM 



ANIMATION DISPLAY PROGRAM STORAGE REGION 
(GAME PROGRAM) 



AUDIO GENERATING PROGRAM STORAGE REGION 
(INCLUDING SOUND DATA PROGRAM) 



GRAPHIC DATA STORAGE REGION 

• ORIGINAL PICTURE DATA 
■ POLYGON DATA 

• TEXTURE DATA 



CREATION IMAGE DATA STORAGE REGION 
(BACK-UP DATA STORAGE REGION) 



451 



,452 



453 



,454 



,455 



456 



READ- 
ONLY 
REGION 
45A 



I WRITABLE 
> REGION 
J 45B 



04/15/2004, EAST version: 1.4.1 



U.S. Patent Jan. 13, 2004 Sheet 4 of 14 US 6,677,967 B2 



FIG. 4 



152. 



154. 



STORAGE REGION MAP IN MAIN BODY RAM 



15 



PROGRAM DATA TRANSFER AND STORAGE REGION 

• OS DATA STORAGE REGION 

• IMAGE CREATING PROGRAM STORAGE REGION 

• ANIMATION DISPLAY PROGRAM STORAGE REGION 

• SOUND DATA STORAGE REGION 

• GRAPHIC DATA (INCLUDING POLYGON DATA, 
TEXURE DATA EXCEPT FOR ORIGINAL 
IMAGE DATA) 

IMAGE CREATING DATA STORAGE REGION 
ORIGINAL IMAGE DATA STORAGE REGION 



. r - „ . . 



; MONOCHROME 
[CAPTURED IMAGE 
IDATA STORAGE REGION! 



COLOR CAPTURED 
IMAGE DATA 
STORAGE REGION 



«■ -i r n i 

CUTOUT IMAGE DATA STORAGE REGION 
1 j 2 j 3 | 4 

U j-..-. i. 1 



15A 



15B 
151 

153 



DISPLAY IMAGE DATA STORAGE REGION 



155 



04/15/2004, EAST version: 1.4.1 



U.S. Patent Jan. 13, 2004 Sheet 5 of 14 US 6,677,967 B2 



F I G. 5 




SELECT DOLL FOR IMAGE CREATION 



Sl(FIG.ll) 



ENTER "CAPTURED IMAGE" MODE (FIG. 6) 



S2(FIG.L1^ 
G.12) 



IMAGE SELECTING PROCESS(FIG.S) 



LS2(FIG.l 

r " 

S3(FIG.12) 



IMAGE CONFIRMING PR0CESS(FIG.7) 



r 



S4(FIG.13) 



CUTOUT( TEXTURE) AREA SELECTING PROCESS 
(FIG. 8) 



.S5(FIG.14) 



CUT0UT( TEXTURE) CONFIRMING PROCESS(FIG.9) 



CUTOUT( TEXTURE) SAVING PROCESS(FIG.LO) 



EXIT FROM "CAPTURED IMAGE" MODE 



PASTE CAPTURED "DESIGN" ON "FACE" 



r 
r 
r 
r 



MODIFY OTHER PARTS TO COMPLETE "TALENT" 



S6 



S7(FIG.15) 



S8 



S9(FIG.16) 



S10(FIGS. 17,18) 



04/15/2004, EAST version: 1.4.1 



U.S. Patent Jan. 13, 2004 Sheet 6 of 14 US 6,677,967 B2 



F I G. 6 



IMAGE CAPTURE/ IMAGE 
SELECT PROCESS START 




INPUT DATA FOR CREATION Mil 




04/15/2004, EAST version: 1.4.1 



U.S. Patent Jan. 13, 2004 Sheet 7 of 14 US 6,677,967 B2 



F I G. 7 



START ^ 



CONVERT INTO COLOR TEXTURE 



-S21 



INPUT CAPTURED DATA M 22 



CONVERT INTO MONOCHROME 
TEXTURE 



— S23 



DISPLAY COLOR TEXTURE 



— S24 



DISPLAY MONOCHROME TEXTURE 




YES 

^~TO STEP~si"^ 



<liET 



RETURN TO STEP S3 



04/15/2004, EAST Version: 1.4.1 



U.S. Patent Jan. 13, 2004 Sheet 8 of 14 US 6,677,967 B2 




04/15/2004, EAST Version: 1.4.1 



U.S. Patent Jan. 13, 2004 Sheet 9 of 14 US 6,677,967 B2 



F I G. 9 




CUT OUT SELECTED AREA — S41 



CHANGE SIZE 



-S42 



WRITE CUTOUT IMAGE DATA 1-S43 
INTO FREE AREA IN STORAGE 
REGION 154 



DISPLAY CUTOUT IMAGE DATA -S44 
IN AREAS IN STORAGE REGION 
154 AS ICONS 




04/15/2004, EAST Version: 1.4.1 



U.S. Patent Jan. 13, 2004 Sheet 10 of 14 US 6,677,967 B2 



FIG. 10 



START 




04/15/2004, EAST Version: 1.4.1 



U.S. Patent 



Jan. 13, 2004 Sheet 11 of 14 US 6,677,967 B2 



FIG. 11 





04/15/2004, EAST Version: 1.4.1 



U.S. Patent Jan. 13, 2004 Sheet 12 of 14 US 6,677,967 B2 

FIG. 13 




A/LEFT -DECIDE 
B/R I GHT- CANCEL 



FIG. 14 



0 










1 ^ 












l«l 


( QBRIGHTNESSD Q CONTRAST D Q TONE D ^ 

(\ | | | | ;<o^ i=i=^=^=l & i iffii ! ! ij 


R 



04/15/2004, EAST version: 1.4.1 



U.S. Patent 



Jan. 13, 2004 Sheet 13 of 14 
FIG. 15 



US 6,677,967 B2 




04/15/2004, EAST version: 1.4.1 



U.S. Patent 



Jan. 13, 2004 



Sheet 14 of 14 



US 6,677,967 B2 



FIG. 17 




FIG. 18 

















f SELECT DOLL ^ 




] ® ■ 





04/15/2004, EAST Version: 1.4.1 



US 6,677,967 B2 



1 



VIDEO GAME SYSTEM FOR CAPTURING 
IMAGES AND APPLYING THE CAPTURED 
IMAGES TO ANIMATED GAME PLAY 
CHARACTERS 

This application is the Division of application Ser. No. 
09/190,601 filed Nov. 12, 1998 , now U.S. Pat. No. 6,285, 
381. 

BACKGROUND OF THE INVENTION 

1. Field of the Invention 

The present invention relates to image display devices 
such as home videogame systems, and more particularly to 
an image display device which captures image data supplied 
from a video source, such as a television receiver, a video 
camera, etc., and combines the captured image data with 
predetermined image data to create and display a combined 
image. The invention also relates to such a system which 
processes an image obtained by the capture process before 
applying the processed image to the predetermined image. 

2. Description of the Background Art 
Conventional image processing software for personal 

computers allows a user to cut out a partial image from a still 
picture and superpose it onto another still picture to create a 
desired image. Such conventional image processing 
software, however, generally requires the user to perform 
complicated operations to accomplish this result. For 
example, it was generally necessary in one such prior 
arrangement for a user skilled in operating personal com- 
puters to perform a sequence of complex image creating 
process steps by, for example, specifying each action cor- 
responding to the processing procedure through operating 
system functions. In such conventional image processing 
techniques, for example, a user might be required to perform 
the operations of: 

reading two still pictures and storing the information into 
a given storage location on hard disk; 

specifying a cutout area on one of the still pictures; 

temporarily saving the cutout image data in a memory; 

specifying a display area on the other still picture for 
superposing the cutout image data; 

writing the cutout image data into the storage region 
corresponding to that display area; 

discarding the unwanted image after cut out; and 

displaying the superposed image. 

It was generally difficult for users untrained in operating 
personal computers to create desired images using such 
steps. Further, even for an operator skilled in operating 
personal computers, the operation was generally so compli- 
cated that creating desired composite images took a long 
time. Accordingly, it was generally quite difficult for chil- 
dren to create such computer graphic images in the course of 
playing a game, for example. Moreover, generally only a 
limited number of images could be combined at a time, and 
displaying the combined image as a moving or animated 
picture was difficult, 

SUMMARY OF THE INVENTION 

The present invention solves this problem by providing an 
image creating arrangement capable of creating a desired 
composite image by very easy and simple operations. 

The present invention also provides an image creating 
device capable of displaying a created image and animating 
it according to a predetermined operating program. 
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One aspect of the present invention is directed to an image 
creating arrangement for capturing image data supplied from 
a video source and combining the captured image data with 
predetermined image data to create a desired image, and 
displaying the desired image on a display device. The 
desired image can be animated in response to user input 
controls and/or program instructions. An example image 
creating device implementation comprises: 

an operation input arrangement used to allow a user to 
input instructions for image creation; 

an initial image data store temporarily storing initial 
image data; 

a captured image store temporarily storing captured image 
data; 

a cut out image store temporarily storing image data cut 

out from the captured image data; 
a display image store storing image data for the desired 

image to be displayed; 
a program store storing a program for image creation; and 
a processor performing an image creation operation on the 

basis of the stored program; 
wherein the stored program includes, 

a first write program instruction sequence responsive to 
operation of the operation input arrangement, for 
capturing the image data supplied from a video 
source and writing the image data into the captured 
image store as captured image data, 
a first read/display program instruction sequence for 
reading the stored captured image data and display- 
ing the captured image on a display device, 
a cutout program instruction sequence for, when the 
operation input arrangement is operated to specify an 
image area to be cut out from the captured image 
displayed on the display device, temporarily storing 
the image data corresponding to the specified area 
into the cutout image store as cutout image data, 
a second read/display program instruction sequence for 
reading the cutout image data stored in the cutout 
image store and displaying the cutout image data as 
an icon or other small image in part of screen of the 
display device, 
a second write program instruction sequence for read- 
ing the initial image data stored in the initial image 
data store and writing the initial image data into the 
display image store, 
a third write program instruction sequence for, when 
the operation input arrangement is operated to select 
the displayed cutout image (e.g., in order to combine 
and display the cutout image in an image display 
position corresponding to a part of the displayed 
initial image), writing the corresponding cutout 
image data stored in the cut out image store into a 
corresponding region of the display image store, and 
a third read/display program instruction sequence for 
reading the combined image data stored in the dis- 
play image store in which the cutout image data is 
written and displaying the combined image data on 
the display device. 
The preferred implementation of the invention provides 
many advantages. For example, it is not necessary to newly 
create an initial image, since an initial image for use as a 
base for image creation can be prepared in advance. The 
process of pasting a cutout image into a specified position 
can be accomplished by the easy operation of specifying the 
position on the displayed original image. Moreover, as it is 
possible to capture the image while viewing the displayed 
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object produced from video signal supplied from a video portion, and may write the display image data into the 

source, an image can be captured not only from a still picture display image store. The third write program instruction 

but also from a moving picture. Further, an image in a sequence writes the cutout image data stored in the cutout 

desired area can be cut out by an easy operation of speci* image store into a storage region in the display image store 

fying the cutout area on the captured image. Moreover, an 5 corresponding to a polygon position of the part to be 

image to be pasted can be easily selected since cutout images changed which was specified by operation of the operation 

can be recorded as icons. In this way, a desired image can be input arrangement. The processor reads the combined image 

created from previously prepared initial original image(s) by data stored in the display image store to display an image 

very easy operations, so that even users unskilled in oper- obtained by combining the image based on the polygon data 

ating personal computers and children can easily create 10 with the cutout image in the display device, and writes 

images. image data changed so that the individual parts based on the 

According to a further aspect of the invention, an image polygon data can be moved at a predetermined cycle or rate 

creating device may provide a program store further con- on the basis of the motion applying program into the third 

taining a correcting program for applying correction and/or temporary store. 

editing processing to the initial image data in a peripheral 15 As stated above, according to this aspect, a three- 
part in the cutout area specified by the operation input dimensional image can be created quite easily. Further, the 
portion before the cutout image data is written into the created image can be displayed as an animated image 
second cutout image store. Thus, when an image in a desired moving according to a predetermined operating program, 
area is cut out from the captured image, its peripheral part According to a further aspect, the program store further 
can be corrected. Accordingly, the original image and the 20 contains a display control program instruction sequence for 
cutout image can be combined without causing unnatural controlling display of the moving object. The display control 
image certificates or other impressions at the boundary program includes a program for displaying a background 
between them. image for the moving object, and a program responsive to 
According to a further aspect provided by the invention, operation of the operation input arrangement, for applying 
an image creating device cutout program instruction 25 changes different from original motion to the motion of the 
sequence cuts out the image data in a rectangular area moving object. A background image can be automatically 
specified by the operation input arrangement, and the cor- created when creating three-dimensional images. 

^ng^^<^^ ^^^m^ ^ DESCRIPTION OF THE DRAWINGS 

fied rectangular area by shading oil a peripheral part around 

a shape defined by excluding the four corners of the speci- 30 These and other objects, features, aspects and advantages 

fied rectangular area. of the present invention will become more apparent from the 

According to a still further aspect, the program store following detailed description of the present invention when 

further contains a motion giving (e.g., animation) program taken in conjunction with the accompanying drawings, 

for applying motion to individual parts of the initial image. \ ^ a perspective view showing an example external 

When reading the combined image data stored in the display 35 appearance of an example image display (e.g., video game) 

image store (e.g., on the basis of the third read/display system according to an embodiment of the present inven- 

program instruction sequence), the processor processes the uon; 

combined image data on the basis of the motion program to F j G 2 ^ a Wock diagram showing an example electrical 

change position and motion of the individual parts of the block diagram of the video garae system shown in FIG. 1; 

combined image. It is thus possible to animate the image 40 3 . g a rf sh an fc m of 

created by combining the captured image wife the original 30 J & ^ ^ an e Qf an Qx[Qm ^ 

image according to a predetermined operating program, ^ medium* 

which enables more enjoyable image creation. m ° , r 

According to a further aspect, the cutout program tern- ™- 4 « a diagram showing an example memory map of 

porarily stores the cutout image data corresponding to the 45 a 15 provided in the game system 10; 

specified area into the cutout image store each time the FIG. 5 is a flowchart showing an example mam routine 

operation input portion is operated to specify an area of used to control operation of an image display device of an 

image to be cut out. The second read/display program example embodiment of the present invention; 

instruction sequence reads the plurality of cutout images FIG. 6 is a subroutine flowchart showing an example 

stored in the cutout image store and separately displays the 50 image capturing/image selecting process; 

plurality of cutout image data parts of the screen of the FIG. 7 is a subroutine flowchart showing an example 

display device. As stated above, according to this aspect, a image confirming process; 

plurality of cutout images stored in the second temporarily pi G g is a subroutine flowchart showing an example 

storing portion are read and displayed in the display device. cutout area selecting process; 

This allows the user to see already recorded cutout images 55 piG 9 ^ a subroutine flowchart show ing an example 

all together to easily select a cutout image to be pasted on the ^ process; 

original image. ^ . a subroutine flowchart showing a cutout image 

According to a further aspect, the initial image store stores . & & 

the initial image data in the form of 3D polygon data savin S P r °cess. 

composed of a group of a plurality of polygons to display an 60 FIG - U 15 a dia S ram lowing an example display pro- 
image of a moving object as the initial image. The program Vlded when a character is selected; 

store further contains a motion applying program instruction FIG. 12 is a diagram showing an example display used in 

sequence for applying motion to individual parts of the the image selecting process; 

object. FIG. 13 is a diagram showing an example displayed used 

The second write program instruction sequence may 65 in the image confirming process; 

create display image data on the basis of the plurality of FIG. 14 is a diagram showing an example display used in 

polygon data stored in the original image data storing the cutout image area selecting process; 



04/15/2004, EAST version: 1.4.1 



US 6,677,967 B2 

5 6 

FIG. 15 is a diagram showing an example display used in example, a Nintendo 64 video game system manufactured 

the cutout image saving process; by Nintendo since 1996) includes an information processing 

FIG. 16 is a diagram showing an example display used in unit (hereinafter referred to as a CPU) 13. The CPU 13 

a cutout image pasting process; performs processing for image creation or game play on the 

FIG. 17 is a diagram showing an example display shown 5 basis of pr°g™m instructions stored in the memory cartridge 

while a desired image is being created (e.g., edited) by using 30 or the disk-like storage medium 45. A bus control circuit 

cutout image* and ^ included in a coprocessor (hereinafter referred to as an 

np 1fl . i r c i RCP) 14 is connected to the CPU 13. The RCP 14 includes 

FIG. 18 is a diagram showing an example of a final « \ • • w r. • i 

... * j * -j- u * ,u * u a first image processing unit (a reality signal processor, 

combined created image providing a character that can be _ . • r. r j . nom ^-i j j • 

animated hereinafter referred to as RSP) 142 and a second image 

1 ' processing unit (reality display processor, hereinafter 

DETAILED DESCRIPTION OF THE refcrrcd t0 * RDP ) 143 ^ RSP 142 * a P^ccssot for 

PREFERRED EXAMPLE EMBODIMENTS executing signal processing such as polygon coordinate 

transformation, light source processing, etc. The RDP 143 is 

FIG. 1 is a perspective view showing the external appear- is a processor for converting polygon data into image data that 

ance of an example image creating (display) system 5 C an be displayed and stored in a data format within frame 

according to an embodiment of the present invention. In this memory included in the RAM 15. The RSP 142, RDP 143, 

example, system 5 comprises a home video game machine and RAM 15 are connected to the CPU 13 through the bus 

system. In FIG. 1, the image creating system 5 of this control circuit 141. The RAM 15 will be described later in 

embodiment captures video signals supplied from a given 20 greater detail referring to FIG. 4. 

video source (a television receiver, a video camera a video M$Q connected t0 the bus comrol circuh 141 are connec . 

cassette recorder, etc ) and processes data by using the image torg u ^ afld a ^ntttikt control circuit 16. The 

based on the captured video signals to create an image suited memory c ^ Mgc (bereinafter referred to „ a cartridge) 30 

to user s preference. fe detachably coonec ted to the connector 12c. Connected to 

In this example, system 20 includes a main unit of the 25 the connector ^d is the disk drive 40. The connectors 121, 

video game system (hereinafter referred to as a game 12 2, 123 and 124 are connected to the controller control 

machine) 10. Formed on the top surface of the housing 11 of circuit 16. These connectors 121 to 124 are connected to 

the game machine 10 are an insertion slot 111 for inserting controllers 20 of the respective users (or players). Further, an 

a cartridge 30, and a power switch 112 and a reset switch 113 aud i 0 ${^1 generating circuit 16a and a video signal 

on the side. The cartridge insertion slot 111 has a cartridge 30 generatmg circuit i$ v are connected to the bus control 

connector 12c (see FIG. 2) provided on its inner side. circuit 141 audio signal generating circuit 16a is 

Provided on the front panel of the game machine 10 are connected to a speaker 51 in the display device 50 through 

controller connectors 121, 122, 123 and 124 for connection an output connector 17a. The video signal generating circuit 

with hand-controllers 20. An extension connector 12d for ig v j s connected to a CRT display 52 in the display device 

connection with a disk drive device 40 is provided on the 35 50 lnrougn m output connector 17v. 
bottom of the game machine 10. 

The controller 20 has a housing 21 with a plurality of Example Memory Maps 
(three in the drawing) handles. Provided on the top of the 

controller housing 21 are: FIG * 3 shows a stora S e re S ion (memory) map of the 

- c . • n c c • j- 40 cartridge 30 and/or disk 45 as an example of an external 

a direction specifying switch 22 for specifying direction t & ,. .... . _ , r ~ . , t , 

c , r l « u- . . 1 u storage medium. When the cartridge 30 is used as the 

of movements of characters or objects operated by a ... _•• • 1 . 1 A *i 

. external storage medium, it includes a non-volatile memory 

p y 1 (hereinafter referred to as ROM) 31 for fixedly storing 

an analog stick (also called a 3D stick or a joy stick) 23 program data and a wri table/readable memory (hereinafter 

for analogously specifying the moving direction, and 4S referred t0 as RAM) 32 ^ R0M 31 mcludes 

a group of operating switches 24 including a reset/start . , /TrVk . e . • m 

button 25 an ic ^ entl ^ catl0n information storage region 311, 

The group of operating switches 24 includes switches 24a an 0S information storage region 312, 

to 24/, which are used to specify actions of the player an image creating program storage region 313, 

characters or player objects in a game mode, and to select 50 an animation display program (game program) storage 

various kinds of commands in an image creating input mode. region 314, 

Side switches 26L and 26R for requesting other actions are an audio generating program, including sound data pro- 
provided on the upper side of the housing 21. slorage reg ion 315, and 

Provided on the top of the housing 41 of the disk drive 40 , . , , t , 

* A-y * u * j * *u * a graphic data storage region 316. 

is a connector 42 to be connected to the extension connector 55 Ct *A ■ tUa in ;„*v™ft™ ctn „„ fl ^ • t . 

^-ij a j- 1 • 1 . a-i ■ c j f . .. c Stored in the ID information storage region 311 in the 

I2d. A disk insertion slot 43 is formed on the front side of Dri . , - 1 om . Mrt - * _ 

. , ... ^ , . 4 . . . ^. . 4 . ROM 31 are the serial No. of the external storage medium, 

the housing 41. Inserted into the disk insertion slot 43 is a vers i on data e tc 

writable/readable disk-like recording (or storing) medium ^ a !? e V j^ 10 ?. V!^/ 0 „■ * \ * • hi 

. - r , , r 1 \ A u Stored m the OS (operating system) storage region 312 

(hereinafter referred to as a disk) 45, such as a magnetic . - . v f !• • j j 

\. , & are information such as graphic microcode, sound 

disk, magneto-optic disk, etc. 60 j ^nn i*u * 

a j- 1 j f n i"i l u u. 1 • • * * microcode, CPU library, etc. 

A display device 50 like a household television receiver is 0i , , . J . . . 

. i_- ia Stored in the image creatmg program storage region 313 

connected to the game machine 10. r r • fu u • .u 

& are programs for realizing the operation shown in the 

Game System Example Block Diagram flowcharts in FIG. 5 to FIG. 10 described later, such as: 

FIG. 2 is a block diagram showing the electric 11 con- 65 write/read/display control program instructions, 

figuration of the video game machine system 10 shown in polygon coordinate calculating program instructions, 

FIG. 1. In FIG. 2, the game machine 10 (which may be, for icon display program instructions, 
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parts select/display program instructions, 

color/brightness control program instructions, etc. 

Stored in the animation storage region 314 are programs 
for giving predetermined action (or motion) by using a 
user-created image (i.e., game program). 

Stored in the audio program storage region 315 are audio 
programs for BGM, sound effects in game, etc. 

Stored in the graphic data storage region 316 are original 
picture (or original image) data, polygon data, texture data, 
etc. 

The RAM 32 is used to store image data for creation 
captured from video signal by the user, or game backup data. 

When the disk 45 is used as the external storage medium, 
the storage region is divided into a read-only region 45Aand 
a writable region 45B, The read-only region 45 A corre- 
sponds to the ROM 31 in the cartridge 30, which is used to 
store the ID information storage region 451, OS information 
storage region 452, image creating program storage region 
453, animation display program storage region 454, audio 
generating program storage region 455, and graphic data 
storage region 456. The writable region 45B corresponds to 
the RAM 32, which is used to store creation image data 
storage region. 

Whether to use the cartridge 30 or the disk 45 as the 
external storage medium depends on the required storage 
capacity (especially the capacity required in the writable 
region). This choice can be freely selected by the software 
developer. 

FIG. 4 is an example storage region memory map of the 
RAM 15 provide in the game machine 10. In FIG. 4, the 
RAM 15 provides a program data transfer and storage region 
15A and an image creating data storage region 15B. In one 
example, part of the program data stored in the ROM 31 or 
the read-only region 45 A in the disk 45 shown in FIG. 3 are 
transferred and stored in the program data transfer and 
storage region 15A. The image creating data storage region 
15B is used to temporarily store image data being processed 
in respective process steps for creating a desired image by 
using a captured image. 

For example, the image creating data storage region 15B 
includes: 

an original or initial image data storage region 151, 
a monochrome captured image data storage region 152, 
a color captured image data storage region 153, 
a cutout image data storage region 154, and 
a display image data storage region 155. 
The original image data storage region 151 is used to store 
original image data (e.g., an initial picture) for image 
creation by the user. In one example, such original image 
data comprises data representing a plurality of polygons 
defining images for display, e.g., an image of the character 
shown in the frame in FIG. 11 (described later), and texture 
(pattern tiles) data to be pasted on to certain parts of such an 
image formed with a plurality of polygons, such as hands, 
legs, body, face, etc. 

The monochrome captured image data storage region 152 
is used to temporarily store video luminance data that may 
vary frame by frame (i.e., moving pictures) when a video 
signal is supplied. 

The color captured image data storage region 153 is used 
to temporarily store color image data of an image to be 
captured from the video (e.g., moving pictures) displayed in 
monochrome. 

The cutout image data storage region 154 is used to 
temporarily store images to be cut out from the captured 
color image data as texture data. This cutout image data 
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storage region 154 stores color data for a plurality of images 
(e.g., a face of a person) cut out from the image captured 
from the video signal, for example. 
The display image data storage region 155 is a so-called 
5 frame memory. It defines a storage region corresponding to 
the number of dots on the display screen of the CRT display 
52. It is used to temporarily store color data displayed on the 
screen. 

10 Example Steps for Creating an Animation from 
Captured Images 

FIG. 5 is a flowchart showing an example main routine 
used to define the operations of the image creating/display 
device of an example embodiment of the present invention. 

15 FIGS. 6 to 10 are flowcharts of subroutines showing details 
of certain steps in the main routine. In FIG. 5, the numbers 
of diagrams enclosed in parentheses within the step boxes 
indicate those flowcharts of corresponding subroutines, and 
the Figure numbers on the right-hand side of the step 

20 numbers indicate corresponding drawings showing 
examples of images displayed in the respective steps. 

The operation of the image creating device of this 
embodiment will be described referring to the main routine 

25 shown in FIG. 5 and the examples of screen displays 
corresponding to the individual steps. First, in step SI, a 
plurality of objects such as, for example, characters such as 
dolls, animals, and the like, are displayed in the CRT display 
52 as starting or initial objects. Such initial objects are 

3Q candidates the user may select for changing the face, clothes, 
etc. The user selects a desired character such as a doll or 
animal. FIG. 11 shows an example of an image displayed 
when a character is selected. At this time, icons showing 
functions selectable by the user are displayed around the 

35 display frame showing the original initial (unmodified) 
image of the selected character. 

Next, in step S2, when the user selects the "Capture 
Image*' icon indicating capturing of image, the image cre- 
ating device enters the capture image mode. In response, 

^ input video supplied from a predetermined video source 
(e.g., a television receiver, a video camera, a videocassette 
recorder, etc.) is displayed in an input video display frame on 
the CRT display 52. Further, commands indicating "Capture 
Image" and "Return" for returning to the display in FIG. 11 

45 are displayed as the next selectable functions below the 
input video display frame (see FIG. 12). With these com- 
mands displayed, the display also indicates which of the 
operating switches 24a and 24b is to be used (A, B), and 
which of the four points (up, down, left, right) of the 

50 direction — specifying switch 22 is to be pressed (left, right) 
to select the commands. At this time, a live moving picture 
which varies frame by frame is displayed in monochrome in 
the input video display frame. 
When the user presses the operating switch 24a or the left 

55 side of the direction — specifying switch 22 to specify "Cap- 
ture Image," the screen shows a display for a process of 
confirming the image (see FIG. 13) in the next step S4. In 
this image confirming process, the captured image is dis- 
played in color in a large area in the center of the screen, and 

60 the live video being inputted is also displayed in mono- 
chrome in a small area in the lower right part of the screen. 
Also, commands are displayed asking the user whether to 
confirm selection of the capture the color image displayed in 
the upper center of the screen or whether to cancel it are 

65 displayed. 

When the user selects a decision command, a process of 
selecting a cutout area as texture is performed in step S5 (see 
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FIG. 14). For example, as shown in FIG. 14, the captured determined that the "Return" command is selected in this 

color image confirmed in step S4 is displayed in a large area, waiting state, the routine returns to the operation in step S2. 

with a box (e.g., or a rectangular box) for specifying the area If it ^ determined in step S14 that the switch 24a (or the 

to be cut out from it. The user can change the size and left side of the ^ nciion specifying switch 22) is pressed, or 

dimensions of this displayed box m arbitrary ways by 5 that the "Decision" command is selected, the routine moves 

operating a cursor shaped like a hand. Controls for adjusting ^ nej£t g <^ 

brightness, contrast, lone, etc. of the captured image data are " 

displayed below the display frame showing the captured Nexl » referring to FIG. 7, the image confirming process 

color image. The levels of brightness, contrast, and tone (step S4) will be described in detail. First, at the instant of 

corresponding to the controls can be changed by operating imzge capture determined in the above -described step S14, 

the cursor. When the user has finished selecting a cutout a process of converting from the video signal of the input 

area, an icon indicating the decision (e.g., "thumbs up") is vide0 int0 ^Iot data is performed (step S21). Next, in Step 

selected. ^22, the color data of the capture color image is written into 

Next,' in step S6, a process for confirming the cutout the stora S e re 6 ion 153 Next > ? n ste P S23 > lhe video si g" al * 

image as texture is performed. Next, in step S7, the cutout 15 converted into monochrome image data (luminance signal) 

image is saved as texture data in a free area in the cutout 10 be displayed as a live video moving picture and written 

image data storage region 154. Then, as shown in FIG. 15, int0 the stora S e re S 10n 152 Next > in ste P S24 : ^ c ° lor 

the selected cutout image texture data is displayed in the captured image data stored m the storage region 153 is 

center of the screen. Previously selected cutout image tex- displayed as a still picture color image as shown in the upper 

ture data are also saved in some area in the image data M frame in nG * 13 ' Next > in Stc P S25 ' th f monochrome 

storage region 154, and displayed as icons or thumbnails on ca P tured ^ stored 10 the stora S e re g 10D 152 ^displayed 

the right side of the screen in a reduced size. At step S8, the io the shown b V the lower frame m nG - 13 » a Uve 

image capturing processing is finished. video movm S P icture * At this time > the svstem als0 Splays 

»t , • OQ ^.^1 . , • a tov(11 „ in the lower left part of the screen that a "Decision" or 

Next, in step S9, the saved selected cutout image texture 1f , 5.,,. , . j . tiL 

, . , n r n . r m .oni,,^ ; man ^ Cancel" command should be selected, to prompt the user to 

data (e.g., ot a race cut out trom the captured image) is 2 s . fjL : C . A . , / . , , 

,. t , • t . u • ci^ nc \i «in.* i select one of them. Then, in step S2o, if it is determined that 

display ed in an enlarged manner as shown in MG. 16. At this W1 _ . . „, , ■. T \. . . 

r r ft tu r ™ f tu # ■ „ Decision has been selected, the routine moves to the next 

time, the user can edit the face of the cutout image as _ „ ___ ™ . . „ . , , . «~ ,„ . 

j ■ j r* t • c i ♦ Step S5. When Decision is not selected and "Cancel is 

desired. For example, icons of a wig, nose, glasses, etc. can < , .. , • . . ~-»-» 

. , . , . .i i \ i • , 4 , not selected either, the operation moves back to step S22. 

be selected, to cover the head with a chosen wig, change the „_ K _ .„ . ' . ^ , . . *. 

. A nf ,/ nr . . „i» M * e ^„ fTL vir it When "Cancel is selected, the routine returns to the image 

shape oi the nose, or to put glasses on the race. MO. 17 30 „L 0 

shows an example image obtained by editing the face in the selecting process in step S3. 

cutout image through user's choice of editing commands Next » referring to FIG. 8, the processing (step S5) for 

and operations. In the next step S10, an image creating selecting the cutout area (in other words, the size of the 

process for editing other parts of the final image (e.g., required texture) will be described in detail. First, in step 

editing based on choice, for changing the clothes, shapes of 35 s31 > the color captured image stored in the storage region 

other parts, etc.) is performed, to complete the image of a 153 is displayed in a large area on the screen, and the box 

character suited to the user's taste, as shown in FIG. 18. for the particular image to be cut out is selected by using the 

Next, details (subroutines) of steps S2, S3, S4, S3, S6 and cursor (see the rectangular box around the face in FIG. 14). 

S7 in the main routine in FIG. 5 will be described. Next > in stc P 11 15 determined whether the icon indi- 

™ . pi j ci „, r v. 1 u , n eating that the cutout image area has been determined was 

The processing in steps S2 and S3 are accomplished by 40 , to , TC . , * iL t . 

' iL . b . • r 1 u ** selected. If it wasn t selected, the routine moves to step S34. 

shown in FIG. 0. In step Sll, with the image 01 the character ,, t - 1 . j ir ■ j . j .u * ^ 

. r *u • a- 1 a u f cancellation was selected. If it is determined that it wasn t 

as a base of .be miage creaaon d.splayed as shown for moves S35 

examplcinnG.il the user selects a command mdiwnng determj ; ed whether of th / color contro H , bars for 

image capturing, and then the routine moves to step S12. In 45 , . , t 4 4 . ' . . . t , Tf . . 

. en ■ J . j- 1 u • u a brightness, contrast, and tone has been selected. If the color 

step S12, in order to display a monochrome image based on * ' ' . , f . 

the video signal, luminance data contained in the color video control bars have been selected, the color data ot the color 

. . * I a a 4 a - * u * * captured image stored in the storage region 153 is corrected 

signal is extracted and converted into monochrome texture .... 1 . j . tL • *u ^ . 

,** j 4 . , • j.r u *jOr adjusted as selected by the user in the next step S36. 

data, and the luminance data of monochrome captured _ , J . . . J . . t t c - i r 

. , tU . * im kt * • + cn Subsequently, the operation moves back to step S3 1. 

image is wntten into the storage region 152. Next, in step 50 ™\ , . Mm , , , 

S13, the monochrome captured image data stored in the If 11 15 determined in step S35 that the color control bars 

storage region 152 is written into the storage area corre- have not been sele f ted ' then « 15 d etermined in step S37 

sponding to the monochrome image display frame shown in whelher * e box for selecting a cutout area has been 

FIG. 12 in the display image data storage region 155, and changed. If it has not been changed, the routine returns to the 

font data for displaying the command of "Capture Image" or 55 operation in step S31. If it has been selected, a process of 

"Return" are displayed below it on the screen. Subsequently, changing the area selecting box is performed in step S38. 

in step S14, it is determined whether the operating switch Subsequently, when the color control and the cutout area 

24a is pressed (or the left side of the direction specifying selecting box have been specified and the icon indicating the 

switch 22 is pressed) — in other words, whether the "Deci- determination is selected, this selection is detected in step 

sion" command is selected. When the decision command is 60 S32 and the routine moves to the next step S6. When it is 

not selected, it is determined in the next step S15 whether the determined in step S34 that the icon indicating cancellation 

operating switch 246 (or the right side of the direction is selected, the routine returns to the image selecting process 

specifying switch 22) for selecting the "Return" command is in step S3. 

pressed. When the "Return" command is not selected, the Next, referring to FIG. 9, the process of confirming the 
routine returns to step Sll and the operations in steps Sll to 65 cutout image (step S6) will be described in greater detail. 
S15 are repeated. The image creating device stays in a When the cutout area has been selected in the above- 
waiting state until some command is selected. If it is described cutout area selecting process (step S5), the color 



04/15/2004, EAST Version: 1.4.1 



US 6,6' 

11 

image in the selected area is cut out in step S41, When it is 
required that the size of the cutout image be changed, a 
process of changing the size (a process of enlarging or 
contracting the section of the image to be cut out) is 
performed in step S42. Next, in step S43, the color data of 
the cutout image is written into a free area in the storage 
region 154. Next, in step S44, the cutout image data stored 
in the respective areas in the storage region 154 are dis- 
played as icons on the right side of the screen as shown in 
FIG. 15, with the latest cutout image being displayed in the 
large area in the center of the screen, At the same time, icons 
for selecting saving or canceling of the latest cutout image 
are displayed on the left side of the screen. Next, in step S45, 
it is determined whether the icon for specifying "Save" is 
selected. When "Save" is not selected, it is determined in 
step S46 whether the icon indicating "Cancel" is selected. 
When neither of the icons is selected, the routine returns to 
the operation in step S44 and the operations in steps S44, 

545 and S46 are repeated. 

When the icon indicating "Save" is selected, it is decided 
in step S45 and the routine moves to the next step S7. When 
the icon indicating "Cancel" is selected, it is detected in step 

546 and the routine returns to the cutout area selecting 
process in step S5. 

Next, referring to FIG. 10, the process of saving the cutout 
image (step S7) will be described in detail. First, in step S51, 
the process of correcting the color captured image stored in 
the storage region 153 is performed, and then the color 
captured image data is transferred and stored into the storage 
region 155. In the process of correcting color captured 
image, when the captured image is a face of a person, the 
face is represented in an egg-shaped area inside the rectan- 
gular box (see FIG. 15). Accordingly, the captured image 
data in the four corners of the rectangular box is shaded or 
masked off. Or, alternatively, color data in two separate, 
distinct points in the face (e.g., forehead and chin) can be 
added and divided by two to obtain the average, and the 
average color data is written in the part of the captured 
image data outside the face outline. Another method of 
correcting the captured face image involves writing color 
data as if a filter or mask is superimposed on the cutout color 
captured image, wherein the filter or mask is transparent in 
the egg-shaped part surrounded by the rectangular box and 
the transparency decreases (i.e., the mask becomes more 
opaque) in areas outward from the periphery of the egg- 
shaped part. The color captured image thus corrected is 
enlarged and displayed in a large area in the center of the 
CRT display 52 (see FIG. 16) and is superimposed onto the 
shoulders of the character to receive the captured facial 
image. 

At the same time, a plurality of icons of textures showing 
previously captured cutout images are displayed around the 
display frame, and various commands are also displayed. 
Next, in step S52, it is determined whether the cutout color 
captured image should be saved in the external storage 
medium, e.g., in the storage region in the RAM 32 (or in the 
writable region 45B in the magnetic disk 45). When the user 
docs not select the icon indicating saving, this is detected in 
step S52 and the routine moves to step S53. The program 
detects in step S53 whether the icon indicating canceling is 
selected, and if it is determined that the icon is not selected, 
the routine moves to the operation in step S51. In this way, 
when none of the commands is selected, the operations in 
steps S51, S52 and S53 are repeated. At this time, if the user 
selects the command indicating saving (or backup storage), 
this is detected in step S52 and color data of the plurality of 
cutout images stored in the storage region 154 are written 
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into the RAM 32 (or the writable region 45B in the magnetic 
disk 45) and the routine moves to the operation in step S8. 

When the icon indicating canceling is selected, this is 
detected in step S53 and the routine returns to the cutout 

5 image area selecting process in step S5. 

Next, with the character having the face cut out and pasted 
onto it as described above being displayed as shown in FIG. 
18, the operation for causing the character to move (i.e., be 
animated) in a desired way will be described. In this case, as 

10 shown in FIG. 18, the face part of the character is changed 
to the cutout color captured image, and textures of clothes 
chosen by the user are pasted thereon. 

Then, as shown in the left side of the display in FIG. 18, 
icons for selecting types of motion of the character are 
displayed. When one of the icons is selected, an animation 
display program (a program for realizing display of motion 
of the character) corresponding to the selected icon is read 
from the storage region 314 in the ROM 31 or from the 

2Q storage region 454 in the magnetic disk 45, and written into 
the transfer and storage region in the RAM 15. Then the 
CPU 13 executes the transferred animation display program 
to calculate coordinate data of the polygons of the character, 
frame by frame, according to previously prepared movement 

25 patterns and animations. Further, textures (pattern tiles) of 
the hands, legs, body, clothes, etc. are pasted on the indi- 
vidual parts of the character on the basis of the calculation 
about a plurality of polygons, and they are written into the 
storage region 155. The texture data written in the storage 

3Q region 155 is read and displayed, to give motion to the 
individual parts of the body of the character as a still object 
shown in FIG. 18. 

The user can play games by using the image created in the 
above-described embodiment. In this case, the user operates 

35 the direction specifying switch 22 or the analog stick 23 to 
specify movement of the created character, and the back- 
ground image is changed by program according to the 
movement of the character. 
While the above-described embodiment provides an 

40 example in which image of a part of the body (e.g., the face) 
of an initial image, e.g., an image of a character, is changed 
to a separately captured color image, it is noted here that the 
present invention can be applied with a variety of modifi- 
cations and changes. For example, the face part can be 

45 changed to a face of an animal, or an original picture of an 
animal can be used and the face of the original picture can 
be changed to a face of another animal. 

While the present invention has been described in detail, 
the foregoing description is in all aspects illustrative and not 

50 restrictive. It is understood that numerous other modifica- 
tions and variations can be devised without departing from 
the scope of the invention. 
What is claimed is: 

1. In a software-controlled home video game machine 
55 system specifically designed for interactive 3D video game 
play, including 3D animated graphics and associated sound 
generation, said home video game machine system includ- 
ing a user-operable hand-held controller having a housing 
and a joystick provided thereon communicating with (i) a 
60 main processor, (ii) a 3D graphics coprocessor connected to 
the main processor for providing at least polygon coordinate 
transformation and light source processing, and (x ii) at least 
one memory including a frame buffer communicating with 
the 3D graphics coprocessor, said at least one memory 
65 storing plural polygon coordinates defining surfaces of 3D 
animated video game characters, the home video game 
machine system playing interactive games based on soft- 
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ware loaded therein, a method for allowing a video game the 3D character using the home video game machine 

player to create animated 3D images from captured 2D system 3D graphics coprocessor. 

image data and interact with the animated 3D images to 4. The method of claim 1 wherein step (b) includes 

provide interactive game play, comprising: allowing the game player operating the user-operable hand- 

(a) providing a captured 2D color image to the home 5 held controller to adjust brightness, contrast and/or tone of 
video game machine system for storage into said at the captured 2D image. 

least one memory; 5. The method of claim 1 wherein step (b) includes 

(b) allowing the video game player to select a portion of allowing the game player operating the user-operable hand- 
the captured 2D color image by operating the user- held controller to add or change portions of the captured 2D 
operable hand-held controller of the home video game 10 image. 

machine system, the selected portion being at least 6. The method of claim 5 wherein step (b) includes 

temporarily stored in said at least one memory com- allowing the game player operating the user-operable hand- 

municating with the 3D graphics coprocessor of the held controller to modify the nose portion of a captured 2D 

home video game machine system; facial image. 

(c) processing, with at least one of the home video game 7. The method of claim 5 wherein step (b) includes 
machine system main processor or the home video allowing the game player operating the user-operable hand- 
game machine system 3D graphics coprocessor, the held controller to modify the ear portion of a captured 2D 
selected 2D color image portion to convert the selected facial image. 

2D color image portion into a color texture; 2Q H ^ method of claim 5 wher ein step (b) includes 

(d) texture mapping, using the home video game machine allowing the game player operating the user-operable hand- 
system 3D graphics coprocessor, the color texture held controller to modify the hair portion of a captured 2D 
obtained from the processed selected 2D color image facial image. 

portion onto a predefined surface of a 3D video game 9. The method of claim 1 wherein step (a) includes 

character defined by plural polygon coordinates stored 1S automatically masking off all but a facial portion of a 

in the at least one memory communicating with the 3D captured 2D image so that the facial portion can be selected 

graphics coprocessor; and j n ste p (b) and stored in the at least one memory. 

(e) animating and displaying, using at least the main 10. The method of claim 1 wherein step (c) includes 
processor and 3D graphics coprocessor, the 3D video converting the captured 2D image to texture data and step 
game character having the applied color texture to 30 (d) includes using the home video game machine system 3D 
provide interactive video game play in response to graphics coprocessor to utilize said texture data to texture a 
manipulation of the user-operable hand -held controller. facial portion of the 3D video game character defined by 

2. The method of claim 1 wherein step (a) comprises polygon data stored in the at least one memory, 
capturing live video from a camera and generating a 2D still 11. The method of claim 1 further comprising allowing 
image from said captured video and storing the 2D still 35 the video game player to edit to a surface of the 3D video 
image in the at least one memory communicating with the game character other than the predetermined surface by 
home video game machine system 3D graphics coprocessor. manipulation of the user operable hand-held controller. 

3. The method of claim 1 wherein step (b) includes 12. The method of claim 1 wherein the video game player 
displaying a cutout window overlaid onto the captured is allowed to edit the predetermined surface of the 3D video 
image and allowing the game player operating the user- 40 game character through operation of the user-operable hand- 
operable hand -held controller to change the position and held controller after the color texture is texture mapped onto 
dimensions of the cutout window so as to select a cutout the predefined surface of the 3D video game character, 
portion of the captured 2D image and store the selected 

cutout portion in the at least one memory for application to ***** 
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